Add the whiteboard interview mode - #72
Conversation
|
Need some time for testing and self-review; will mark this PR as ready for review once it's set. |
|
Perhaps we will need to record the state of the whiteboard at each stage so that we can provide more information to the user at the end of the interview. |
|
Additionally, pseudocode should be graded separately rather than being combined with optimization like it is currently. |
|
On the concern from #65 about whether Gemini can read a drawn board reliably: one option worth considering is Excalidraw as the board layer. The drawing tools are not the point; the data model is. An Excalidraw scene is a list of typed elements, and boxes, arrows (bound to the boxes they connect) and text keep their actual strings. So next to the JPEG, the agent can get a plain-text summary of the board, e.g. I checked whether it fits the vendoring rules, following the
The costs:
Only the board layer changes. The byte stream, |
|
Hi, @alanhc Thanks for the discussion! Excalidraw was also brought up in the discussions under the Facebook post back then. After looking into it, I think it seems like a solid option to consider, but I also have a few thoughts. If our goal is to simulate an actual interview environment as closely as possible, would Excalidraw make things too convenient for the user? After all, in a real interview, you only have a marker and a physical whiteboard. That raw experience is precisely what we want to deliver in this mode—allowing users to practice explaining their problem-solving approach and thought process purely through drawing on a blank board. Currently, a tester, @Eason0729 , has tested this feature (the non-Excalidraw version) and provided a lot of feedback. Here is a brief summary of the key concerns raised from the testing:
For now, I'll focus on addressing the points raised in the feedback first. As for Excalidraw, we can discuss it further as a potential future improvement. Overall, I actually think this is quite a promising direction. However, my primary concern is that users should have an experience that feels as close to reality as possible; we probably shouldn't compromise on that just to make recognition easier for the model. WDYH |
2228cf7 to
a124a50
Compare
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
a124a50 to
2f6495a
Compare
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
2f6495a to
969a4bb
Compare
|
Agreed, realism should win for this mode. The point is practicing with a marker and a blank board, and shapes plus a text tool would take exactly that away, so I would drop Excalidraw as the board here. If it comes back at all, it would be as some separate "diagram" mode, not as the whiteboard. On the two points from testing:
Happy to help test the touch path once it is in. |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
c6fa1f4 to
9f0559b
Compare
|
Clearing the board between steps makes |
9f0559b to
a0d6bdb
Compare
a0d6bdb to
1947b11
Compare
|
Measure the token cost of the whiteboard with the tooling from #203. Run a whiteboard interview and a regular one on the same problem, then compare them with |
|
Measured on six real interviews against
Per text, with
Board frames are the part that grows. Each one is 60 image tokens on the Live socket regardless of content, and it stays in the context: by the end of each whiteboard session 8 to 10 boards were held, 480 to 600 tokens on every later generation until the sliding window trims them. Across a session that was 2.9 to 4.2% of the prompt tokens. A board goes out at most once a second after a one-second settle, or every four seconds while drawing continuously. |
|
The results look fine in terms of cost. |
Good. Append the above report in |
A whiteboard interview takes the same problem bank and the same six steps and swaps the editor and the test runner for a board. Nothing runs, so correctness is what the candidate can defend by tracing an example across their own drawing. The board travels as a LiveKit byte stream rather than on a data topic, because one board is tens of kilobytes and a data packet carries fifteen, and it reaches Gemini the way a camera frame does, as a realtime image. A tool response is JSON and cannot carry a picture, so read_board asks for the board to be sent again and answers with what an image cannot say: how much is on it and how long ago it was drawn. A camera frame on that stream would displace the board, so a whiteboard interview forwards no camera even where the operator enabled it. Bundle 29, live prompt 21, follows the surface. A whiteboard session is told it has no editor and no test runner, is offered read_board in place of read_editor, and records board_snapshot where the other records an editor snapshot or a test event; neither may record the other's source. The phases about written work are gated on strokes instead of on characters, and Test on the cases named against the drawing instead of on a run, or an interview with no editor would be refused the second half of its own flow. The report still reads the transcript alone and the replay still carries no board.
The whiteboard was live but nowhere else: the report was written from a transcript and an editor nobody opened, the recording kept no drawing, and the checklist beside the timer called step four Coding while the interviewer was asking the candidate to trace. The board now rides the report request as an inline image, ahead of the brief and on every repair, and the brief says what it is looking at: nothing ran, so correctness is the trace the candidate walked, and the phases keep their names with Coding meaning that trace, Test the cases named against the drawing, and Optimizations the complexity they confirmed. The system instruction is left as it is, so every report call still shares its prefix, and the brief tells a whiteboard reviewer how to read the rules that speak of code. That is report prompt 17, inside the same bundle 29, and the reviewer is told when no board arrived rather than sent looking for an attachment that is not there. The recording keeps the drawing as the operations that made it. One board as a JPEG is past the per-event ceiling on its own and would spend the whole per-interview budget on a handful of frames; the same board as strokes is a few kilobytes, so the replay page and the recording template redraw any moment of the interview with the module the candidate drew on, rather than the few moments a photograph could afford.
A final board cannot represent work cleared between phases. Keep one JPEG checkpoint per phase for grading and a vector marker for replay, so review survives a clear without putting images into the replay budget. Clear itself becomes one undoable step: with the earlier phases kept, an accidental clear is the costliest mistake left at the board. The checkpoints are why clearing is safe, and the gate on the phases about written work has to agree: it read the newest board's stroke count, so a clear refused every later Coding, Test and Optimizations record. It now asks whether any board carried a drawing, and the count leaves out erasures, which leave no ink. A board stream that never finishes gives its read slot back after a deadline, and the image read_board asks for reaches Gemini ahead of the tool response, which Gemini starts answering as soon as it arrives.
One fixed width was either too wide to take out a character or too narrow to clear a region. Three offered sizes cover both, and a fixed list rather than a slider keeps every width on the replay one a button can make. The cursor becomes a circle the size of the erasure.
A whiteboard report read as an editor one: a Coding score, REACTO step names the candidate was never shown, and one board folded at the foot of the card. The page already captured the board as each step closed, for the grader, and kept none of it. The report now opens on the board, step by step: the image as that step closed, its score and what the interviewer wrote down, then the final board. Steps, plans and evidence take the whiteboard names, and the score is Board work. Only the display changes; the report keeps the REACTO ids it is graded against. The images are never saved, so a report reopened from history says where they are instead. Those step images only exist if every board reaches the agent before the report is frozen. The board stays locked until the room is joined, so the replay and the interviewer see its first stroke. A reconnect sends boards ahead of a queued end_interview, an end mid-stroke closes the stroke and captures the phase parked behind it, and the end waits for uploads only briefly. A checkpoint that failed to upload is retried ahead of every later board, not only on a reconnect, and a failed write still closes its stream. Steady drawing sends a board every few seconds instead of waiting for a pause that never comes.
The board threw away every pointer that was not a mouse, so a tablet could not draw on it at all. Pointer events already carry all three, and the canvas already sets touch-action: none, so the board now takes any of them, one pointer at a time. A hand resting on a tablet reports as touch: a touch is refused while a pen is down, and a pen that lands after a palm cancels the palm's stroke rather than leaving it drawn. A stroke that begins also cancels the settle timer the previous one started, so a board is never exported half-drawn.
A board reaches the interviewer as an image of handwriting and sketches, and a box or arrow it cannot read is one it fills in with a guess. The candidate narrates while drawing, and that speech is already in the conversation, so a whiteboard session is told to read an unclear mark by what was said while it was drawn, and failing that to ask what it stands for, as a person at the board would. The board stays raw and the live prompt keeps version 21, which has not shipped.
The README said what the whiteboard sends and where, but not how a candidate picks it, draws on it or reads its report, so a first try had to discover the toolbar, the input rules and the renamed steps alone. A section now walks through choosing it in the lobby, when the board takes strokes, pen and touch input, each toolbar control, what Jim sees and when, the six board steps, and what the report shows.
Measured on six real interviews, three per surface, so a reviewer can tell the fixed overhead from the part that grows: the opening is 23 tokens below the editor's, and each board sent adds 60 image tokens that every later generation is billed on again.
1947b11 to
03d0222
Compare
added in 03d0222 |
|
Thank @ColtenOuO for contributing! |
3 participants
Summary
A whiteboard interview is the coding round held without a compiler. The lobby offers it next to the duration and the loop. The editor, the starter code and the test runner are replaced by a drawing surface, and the candidate shows correctness by drawing an approach, tracing an example across it, and naming the cases that would break it. It uses the same problem bank, the same six REACTO phase ids, the same rubric and the same report schema. Only the surface and what each step asks for change.
Closes #65, with two deliberate departures from the issue's proposal:
What the candidate sees
repeatexamplealgorithmcodingtestoptimizationsHow a board reaches the interviewer
The browser keeps the drawing as vector strokes (
web/whiteboard.js). That is what makes undo and replay possible. Rendering a board to pixels happens only when something needs an image.flowchart LR Stroke["Stroke ends"] --> Settle["1 s with no new stroke"] Settle --> JPEG["Canvas exported as JPEG<br/>quality 0.72"] JPEG -->|"LiveKit byte stream<br/>topic board_image"| Drain["Agent drain task<br/>bounded at 512 KiB"] Drain -->|"queue of 2"| Loop["Room loop: pump_board"] Loop --> Latest["Latest board in memory"] Loop -->|"at most one per second<br/>realtimeInput.video"| Live["Gemini Live"] Tool["read_board tool call"] --> Resend["Resend latest board"] Latest --> Resend Resend --> Livestrokescount and, for a phase checkpoint, acheckpointid.read_boardanswers with it.read_board. Returns JSON with the stroke count, the board's age and the timer, then resends the latest image, because a tool response cannot carry a picture.What the interviewer is told
There is one live prompt. The sentences that name the surface sit in a
Surfacestruct with a coding and a whiteboard value, so a rule cannot end up in one prompt and not the other. Rebased onto #203, #192 and #202: the coding prompt is byte-identical to main's, and only the whiteboard entries are new intests/golden/prompts.json. The thinking-time rule from #192 says "board changes" rather than "editor changes" at a board.read_boardin place ofread_editor. It is told nothing will run.board_snapshotis a valid evidence source only at a board, and Test is recorded from the cases named against the drawing.Phase checkpoints and grading
The report reviewer sees the board as it stood when each step closed, not just the final one. A candidate who clears the board between steps does not lose that evidence.
sequenceDiagram participant B as Browser participant A as Interview agent participant G as Gemini report model A->>B: framework_state (phases with evidence) B->>B: New phase ids, in REACTO order B->>A: JPEG + checkpoint id (after pointerup if mid-stroke) A->>A: Validate the id, keep one image per phase B->>A: Final JPEG when End is pressed B->>A: end_interview, once the uploads finish A->>G: generateContent(labelled inline JPEGs + prompt) Note over A,G: Retries and schema repairs resend every image G-->>A: Structured JSON report A-->>B: Report, stamped with interviewModeframework_stateupdate is reduced to known phase ids that have not been captured yet. A phase that completes while the pointer is down waits for pointerup, so the image and the replay describe the same stroke.end_interviewis sent only after they finish. That way the report is never frozen on an older board.Replay and recording
Replay stores no JPEGs. A board as a JPEG is over the per-event size limit and would use up the per-interview replay budget within a few frames. Replay instead stores the operations that drew the board:
stroke,undo,redoandclear, in batches under 24 KiB, each optionally tagged with a checkpoint id. The replay page and the recording template rebuild any moment of the board with the same module the candidate drew on (createBoard+applyOp). Every operation is validated on the way back in: colour, width, point count and coordinates.The report card
A whiteboard report opens on the board, below the committee summary:
.mdfile lists the steps and their evidence without base64.Whiteboard